Review of Recent Distillation Studies

Knowledge distillation has gained a lot of interest in recent years because it allows for compressing large deep neural network (teacher DNN) into smaller DNN (student DNN), while maintaining its accuracy. Recent improvements have been made to knowledge distillation. One such improvement is the teaching assistant method. This method involves introducing an intermediate "teaching assistant" model between teacher and student. The first trained mimic teacher, then student assistant. multi-step process can improve performance. Another curriculum gradually training by exposing increasingly difficult concepts over time, similar learning humans. help learn more stable consistent manner. Finally, there mask Here, specifically attention mechanisms learned not just overall output DNN. These enhance enable creation efficient DNNs.